Prepare Worker for External DF SDK support (and a few DF bug fixes) #746

davidmrdavid · 2022-01-29T03:22:10Z

Problem Statement

The Durable Functions (DF) SDK for PowerShell is currently tightly-coupled with the worker code. This results in two major problems:

SDK improvements and patches cannot be released quickly, they're slowed down by the worker-release infrastructure
Users cannot choose which version of the SDK they want to use, which makes breaking changes harder to implement.

Solution

This PR refactors the DF-initialization logic in the worker to allow for users to take a dependency on a DF SDK downloaded from the PowerShell gallery (i.e an external SDK). This sets the foundation to extract the DF SDK from the worker, thereby removing this tight-coupling issue.

In the future, we plan to remove the built-in DF SDK implementation in the worker so that using the external SDK becomes the default option. However, this would be a breaking change and so we have no plans to remove it in the short term.

Caveat: While this PR contains the logic to enable using the external SDK, the specific logic to enable its usage will remain commented out until the external SDK is released on the PowerShell gallery. Still, merging these changes soon will allow us to ensure that the refactoring we've done is safe and doesn't break existing internal-SDK users.

Other goals

In addition, this PR also fixes several known DF SDK issues. These are listed below:

These fixes should have probably deserved their own PRs. However, due to time-pressure, they ended up being part of the same branch.

Description of Changes

Changes to enable the use of an external DF SDK

The Durable directory in the worker has been split into two folders: DurableSDK and DurableWorker. The former represents the SDK-specific components that, in the future, could be removed from this repo; this means it also represents the SDK-components that ought to exist in the external SDK. Meanwhile, the DurableWorker directory represents DF logic that is intrinsic to the worker, and ought to stay in this repo.
The new DurableSDK directory in particular is meant to be "drag and dropped" into an external SDK implementation; it's self-contained. As a result, we have removed any coupling of files within it with utility classes elsewhere in the repo. In practice, this means a few function definitions have been "duplicated" so that they now exist independently inside the "DurableSDK" directory. This is fine since we plan to eventually remove the DurableSDK repo from the worker.
The OrchestrationInvoker.Invoke method, where orchestration replay starts, now explicitely checks for whether an external DF SDK is enabled. If that were the case, it defers the driving of orchestration replay to the external SDK. It does this by invoking its "externalInvoker" reference.
In order to initialize the DF binding uniformly across the external and internal SDKs, we've delayed some of the binding initialization logic. In particular, the operations pwsh.AddParameter(orchestrationBindingInfo.ParameterName, ...) and pwsh.TracePipelineObject now occur inside the OrchestrationInvoker instead of in the Functions-general Invoke method. This allows the external SDK's Invoke method to control what data it passes to orchestrationBindingInfo.ParameterName.

Other changes

This PR also includes a few urgent bug fixes of the DF SDK. See them below.

We now expose a Get-DurableTaskResult utility which allows users to get the result of already-completed Tasks. This is needed because, if a user didn't immediately bind a task's result to a variable when "awaiting" it, they would previously lose access to it. The code for this was contributed by a community member here: Allow completed Tasks to expose their results #753.
Previously, WaitForExternalEvent tasks did not output its results to users. This has been fixed.
Previously, the fields InstanceId and IsReplaying where not exposed to users. Now they're available.

Future plans

After merging and releasing this PR, we'll look to continue the development of the external SDK, add an automated-CI to it, and eventually release it to the PowerShell Gallery.

co-authored to an equal extent with @michaelpeng36

…or this logic for the worker to continue working with old SDK

Co-authored-by: Michael Peng <[email protected]>

…rking' into df-worker-and-sdk

…hell-worker into michaelpeng/fix-building

davidmrdavid · 2022-03-14T17:19:13Z

src/DurableSDK/Commands/GetDurableTaskResult.cs

+    using System.Management.Automation;
+    using Microsoft.Azure.Functions.PowerShellWorker.Durable.Tasks;
+
+    [Cmdlet("Get", "DurableTaskResult")]


This is the community-contributed CmdLet that allows users to get the result of an already-completed Task. In other PLs, users can easily do this by re-await/re-yielding that Task, but the PS programming model doesn't have that. The implementation of this CmdLet isn't as efficient as it could be, but I don't think that's something we can't address after merging it.

This addition seems to be independent from the main purpose of the PR. If this is not very difficult, I recommend separating this into its own PR. Unless this is indeed too difficult.

davidmrdavid · 2022-03-14T17:21:11Z

src/DurableSDK/Commands/InvokeDurableActivityCommand.cs

-            var loadedFunctions = FunctionLoader.GetLoadedFunctions();

            var task = new ActivityInvocationTask(FunctionName, Input, RetryOptions);
-            ActivityInvocationTask.ValidateTask(task, loadedFunctions);


These two were removed since they're incompatible with the external SDK. In addition, other DF SDKs don't perform this validation steps. If we want them, they ought to exist outside the DF SDKs.

I do agree that these are useful validations to have though, but if we perform these checks in the DF SDK components, then we're creating a strong dependency with worker-specific utilities.

The intent of ActivityInvocationTask.ValidateTask is to provide user-friendly error messages when the specified activity function does not exits (for example, because of a typo in the name). Without this code, the error messages are really confusing and unhelpful. If you remove it, we need an adequate replacement.

davidmrdavid · 2022-03-14T17:21:31Z

src/DurableSDK/DurableTaskHandler.cs

+                        case HistoryEventType.EventRaised:
+                            var eventRaisedResult = GetEventResult(completedHistoryEvent);
+                            if (eventRaisedResult != null)
+                            {
+                                output(eventRaisedResult);
+                            }
+                            break;


This allows WaitForExternalEvent tasks to output a result

davidmrdavid · 2022-03-14T17:22:29Z

src/DurableSDK/DurableTaskHandler.cs

+        public void GetTaskResult(
+            IReadOnlyCollection<DurableTask> tasksToQueryResultFor,
+            OrchestrationContext context,
+            Action<object> output)
+        {
+            foreach (var task in tasksToQueryResultFor) {
+                var scheduledHistoryEvent = task.GetScheduledHistoryEvent(context, true);
+                var processedHistoryEvent = task.GetCompletedHistoryEvent(context, scheduledHistoryEvent, true);
+                if (processedHistoryEvent != null)
+                {
+                    output(GetEventResult(processedHistoryEvent));
+                }
+            }
+        }
+


Part of the community-contributed logic for "Get-TaskResult". Again, this could use some caching, but I'd rather optimize that later since this is feature is urgently missing.

davidmrdavid · 2022-03-14T17:23:15Z

src/DurableSDK/DurableTaskHandler.cs

            }
            return null;
        }

+        public static object ConvertFromJson(string json)


This used to exist in TypeExtensions, creating a tight-dependency with worker utilities. It's been copied here to allow this folder to be readily copied out into an external SDK

* Define a contract for the external invoker * Remove extraneous comments and variables

…hell-worker into df-worker-and-sdk

davidmrdavid · 2022-07-12T23:06:46Z

Closing in favor of #839

davidmrdavid and others added 30 commits January 28, 2022 19:20

separate DF SDK classes from DF worker classes

4423d13

fix typo

9d73110

DurableSDK now compiles by itself

2fcbfae

Allow ExternalSDK to handle orchestration

f43f350

document next steps

092e333

allow external SDK to set the user-code's input. Still need to refact…

506e4f7

…or this logic for the worker to continue working with old SDK

add import module

7a38ec8

supress traces

9b2cf42

avoid nullptr

47caee9

pass tests

1010f97

fix E2E tests

d916d58

develop E2E tests

946ea37

Enabled external durable client (#765)

64b6c49

Co-authored-by: Michael Peng <[email protected]>

bindings work

9e7cf96

conditional binding intialization

92f113e

conditional import

e5843ce

Added exception handling logic

2ee6b96

Ensure unit tests are functioning properly

ae28f6e

Corrected unit test names

94f995d

Turned repeated variables in unit tests into static members

96f6ee7

Revert durableController name to durableFunctionsUtils

1cb30b8

Merge remote-tracking branch 'origin/michaelpeng/get-durable-tests-wo…

ff5102e

…rking' into df-worker-and-sdk

Fixed issue with building the worker

b8c6a6a

Fix E2E test

baae3cc

Fixed unit test setup

6feb7fe

Fixed another unit test setup

94d83ae

Remove string representation of booleans

d11a561

patch e2e test

0e0c077

remove typo in toString

1c4a7ae

Merge branch 'dev' of https://github.com/Azure/azure-functions-powers…

dc0ed38

…hell-worker into michaelpeng/fix-building

davidmrdavid added 4 commits March 9, 2022 09:32

parametrize test

3d1a7b4

patch new e2e test

adec8fa

patch external contrib

8d5d9bf

fix typo in test

8351653

davidmrdavid commented Mar 14, 2022

View reviewed changes

davidmrdavid and others added 7 commits March 14, 2022 10:30

comment changes

7232e58

Adds IExternalInvoker (#776)

d9b3baf

* Define a contract for the external invoker * Remove extraneous comments and variables

rename hasOrchestrationContext to hasInitializedDurableFunction

edf3011

remove outdated TODO comment

aca4c0b

remove now unused function - CreateOrchestrationBindingInfo

e5ca2ba

Allow worker to read results directly from the external SDK (#777)

f69b146

comment out external SDK path

4387799

davidmrdavid changed the title ~~[WIP] Separation of DF SDK code from Worker code.~~ Prepare Worker for External SDK support Mar 17, 2022

davidmrdavid marked this pull request as ready for review March 17, 2022 17:50

davidmrdavid requested a review from AnatoliB March 17, 2022 17:50

davidmrdavid changed the title ~~Prepare Worker for External SDK support~~ Prepare Worker for External DF SDK support (and a few DF bug fixes) Mar 17, 2022

davidmrdavid requested a review from Francisco-Gamino March 17, 2022 17:50

This was referenced Apr 7, 2022

Fix various DF bugs #783

Merged

Rename DF components #793

Merged

davidmrdavid added 3 commits July 12, 2022 10:36

change serialization

a9a0a6e

Merge branch 'dev' of https://github.com/Azure/azure-functions-powers…

12ccdee

…hell-worker into df-worker-and-sdk

restore orchestrationInvoker and powershellServices

5f862e6

davidmrdavid closed this Jul 12, 2022

davidmrdavid deleted the df-worker-and-sdk branch July 12, 2022 23:06

davidmrdavid mentioned this pull request Jul 18, 2022

Enable external DF SDK #839

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prepare Worker for External DF SDK support (and a few DF bug fixes) #746

Prepare Worker for External DF SDK support (and a few DF bug fixes) #746

davidmrdavid commented Jan 29, 2022 •

edited

Loading

davidmrdavid Mar 14, 2022 •

edited

Loading

AnatoliB Mar 18, 2022

davidmrdavid Mar 14, 2022

AnatoliB Mar 21, 2022

davidmrdavid Mar 14, 2022

davidmrdavid Mar 14, 2022

davidmrdavid Mar 14, 2022

davidmrdavid commented Jul 12, 2022 •

edited

Loading

Prepare Worker for External DF SDK support (and a few DF bug fixes) #746

Prepare Worker for External DF SDK support (and a few DF bug fixes) #746

Conversation

davidmrdavid commented Jan 29, 2022 • edited Loading

Problem Statement

Solution

Other goals

Description of Changes

Changes to enable the use of an external DF SDK

Other changes

Future plans

davidmrdavid Mar 14, 2022 • edited Loading

Choose a reason for hiding this comment

AnatoliB Mar 18, 2022

Choose a reason for hiding this comment

davidmrdavid Mar 14, 2022

Choose a reason for hiding this comment

AnatoliB Mar 21, 2022

Choose a reason for hiding this comment

davidmrdavid Mar 14, 2022

Choose a reason for hiding this comment

davidmrdavid Mar 14, 2022

Choose a reason for hiding this comment

davidmrdavid Mar 14, 2022

Choose a reason for hiding this comment

davidmrdavid commented Jul 12, 2022 • edited Loading

davidmrdavid commented Jan 29, 2022 •

edited

Loading

davidmrdavid Mar 14, 2022 •

edited

Loading

davidmrdavid commented Jul 12, 2022 •

edited

Loading